Using SAS/STAT to implement a multivariate adaptive outlier detection approach to distinguish outliers from extreme values

Hawkins (1980) defines an outlier as “an observation that deviates so much from other observations as to arouse the suspicion that it was generated by a different mechanism”. To identify data outliers, a classic multivariate outlier detection approach implements the Robust Mahalanobis Distance Method by splitting the distribution of distance values in two subsets (within-the-norm and out-of-the-norm), with the threshold value usually set to the 97.5% Quantile of the Chi-Square distribution with p (number of variables) degrees of freedom and items whose distance values are beyond it are labeled out-of-the-norm. This threshold value is an arbitrary number, however, and it may flag as out-of-the-norm a number of items that are actually extreme values of the baseline distribution rather than outliers. Therefore, it is desirable to identify an additional threshold, a cutoff point that divides the set of out-of-norm points in two subsets extreme values and outliers. One way to do this – in particular for larger databases – is to Increase the threshold value to another arbitrary number but this approach requires taking into consideration the size of the dataset since size will affect the threshold separating outliers from extreme values. A 2003 article by Gervini (Journal of Multivariate Statistics) proposes “an adaptive threshold that increases with the number of items n if the data is clean but it remains bounded if there are outliers in the data.” In 2005 Filzmoser, Garrett and Reimann (Computers & Geosciences) built on Gervini’s contribution to derive by simulation a relationship between the number of items n, the number of variables in the data p and a critical ancillary variable for the determination of outlier thresholds. This paper implements the Gervini adaptive threshold value estimator using PROC ROBUSTREG and the SAS ChiSquare functions CINV and PROBCHI, available in the SAS/STAT environment. It also provides data simulations to illustrate the reliability and the flexibility of the method in distinguishing true outliers from extreme values.